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Background:  Critical  Needs  of  Automation, 
Validation,  and  Discovery  for  DoD  Acquisition 
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=  Sponsor  Activity 


1=  JCIDS  Document 


JCIDS  Process  and  Acquisition  Decisions 


OA 


=  Acquisition  decision 


(J-8  CJCSI  3170.01G)(JCIDS,  2009) 


Multiple  Portfolio  Views: 

•  Systems  vs.  Capabilities 

•  Investment  vs.  Capabilities 

•  System  Context 

•  Highly  dependent  programs 
(Joint  Enablers) 

•  Procurement  Optimization 

•  S&T  vs.  future  needs 

•  Sustainment  Efficiency 

•  Market  Value 


•  Data  are  too  voluminous,  unformatted, 
and  unstructured! 

•  Need  to  leverage  automation 

•  Extract  relations  among  PE,  MDAP, 
and  AC  ATII 

•  Extract  costs 
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Acquisition  Research  Program:  Creating  Synergy  for  Informed  Change 
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Methods:  System  Self-awareness  (SSA)  VV 
and  Lexical  Link  Analysis  (LLA) 


•  System  Self-awareness 

-  The  cognitive  interface  between  decision  makers  and  a  complex  system, 
expressed  in  a  range  of  terms  or  -  “features”  -  a  specific  vocabulary,  or 
“lexicon,”  to  describe  the  attributes  and  surrounding  environment  of  the 
system. 

-  Complex  system’s  ability  to  assess  itself  within  a  global  context 

•  Authority 

•  Expertise 

•  LLA  is  a  Text  Analysis  method  using  bi-gram  co-occurrence  word  pair  networks 

-  We  explored  an  analytic  and  visualization  of  Lexical  Link  Analysis  (LLA),  to  link 
warfighter  requirements  with  the  acquisition  programs  and  program  elements 

•  Gallup,  MacKinnon,  Zhao,  Robey  &  Odell,  2009; 

•  Zhao,  Gallup  &  MacKinnon,  2010,  2011a,  2011b,  2011c,  201  Id,  2012a,  2012b, 


2013; 

•  Zhao,  Brutzman  &  MacKinnon  2013 


Comparing  Two  Systems  using  LLA 


DISE 


Unique  Features  of 
System  2 


Lexical  Links  between 
System  1  and  2 


Unique  Features 
System  1 


Discovering  Themes 
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Comparing  Categories  and  Time  Points 
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How  LLA  Methodology  Can  Help 
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LLA  automates  the  possibility  to 
develop  awareness  of  the  “fit” 
between  PE’s,  budget  and 
warfighter  requirements. 


RDTE  Program  Elements 
(DOD  Budget  $$$ 
Justification) 


How  to  validate  LLA? 

Do  PEs  or  Programs  match  requirements? 

Do  inter-connected  PEs  or  Programs  cost  more? 


Weapon  Book 

(Final  Products  for 
Procurement) 
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The  Acquisition  Visibility  Portal  (AVP) 
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•  DoD  acquisition  communities  and  professionals  needs  to  access 
authoritative  and  accurate  data  services  for  decision-making 

-  The  Acquisition  Visibility  Portal  (AVP)  was  such  a  data  service  that 
achieved  this  purpose  by  interfacing  with  Defense  Technical  Information 
Center  (DTIC) 

-  Program  Elements:  http://www.dtic.mil/descriptivesum/] 

-  Warfighter  requirements:  http:/www. dtic.mil/doctrine]). 

-  Defense  Acquisition  Management  Information  Retrieval  (DAMIR; 
http://www.acq.osd.mil/damir/) 

•  Milestones,  costs,  schedules,  and  performance  data  of  selected 
acquisition  reports  (SAR) 

•  Acquisition  Strategy  Reports  (ASR) 


*  i 

Gaps  and  Inconsistencies  of  AVP  ***** 

Data  Sources 


•  The  Office  of  the  Secretary  of  Defense  (OSD)  staff  review  to  determine  if 
the  program  is  properly  prepared  for  the  next  milestone. 

-  Thoroughly  review  these  artifacts  limited  on  staffing  and  little  time 

-  Each  functional  community  reviews  only  the  particular  document  for  which  it  is  responsible 

-  The  systems  engineering  community  typically  only  examines  the  systems  engineering 
plans  (SEP) 

-  The  test  and  evaluation  community  looks  only  at  the  Test  &  Evaluation  Master  Plans 
(TEMP) 

-  The  acquisition  community  looks  at  the  Acquisition  Strategy  Reports  (ASR). 

•  Milestone  documents 

-  Divergent  naming  conventions  may  indicate  the  documents  were  developed  in  isolation. 

-  Meaningful  linkages  between  these  reports,  e.g.,  a  capability  defined  in  the  acquisition 
strategy,  should  be  elaborated  in  the  systems  engineering  plan;  the  testing  of  which  should 


be  described  in  the  TEMP. 

-  Inconsistencies  among  these  documents  may  reflect  a  risk  to  the  program. 


DISE 


Where  Do  the  Gaps  Come  From? 

(e.g.  Compare  ASR  and  TEMP) 


•  What  are  the  features  or  clusters  of  features  (e.g., 
themes)  discussed  in  ASR  but  not  discussed  in 


TEMP? 


•  Reasons  for  gaps 


1 .  A  data  quality  issue  (e.g.,  a  mishandling  of  data  by  AVP), 

2.  A  data  classification  issue  (e.g.,  unclassified  data  vs. 
classified  data), 


3.  A  real  requirement  gap  (i.e.,  a  concept  required  by 

acquisition  for  which  no  engineering  feasibility  document  or 
blueprint  can  be  located). 
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How  Might  LLA  Help? 
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•  Examine  large  collections  of  documents  for  many 
programs  in  various  categories  across  the  acquisition 
and  engineering  communities  by 

-  Detecting  data  quality  such  as  inconsistency,  gaps,  or  bad  data  among 
categories 

-  Identifying  data  dependencies  that  might  be  indicators  for  program  or 
investment  performances  and  risks 

-  Learning  from  the  actual  data  to  see  how  the  common  concepts  are 
expressed  in  different  artifacts  and  communities. 

-  Conducting  a  pair-wise  comparison  exposes  significant  disconnections 
between  them. 


-  Discovering  disconnection  or  gaps  that  could  be  fed  back  to  the  human 
analysts  or  decision-makers  to  perform  further  investigations. 


Data  Access 


DISE 


•Acquisition  Visibility/DMAIR  Portal 

(https://ebiz.acq.osd.mil/DAMIR/PortalMain/DannirPortal.aspx) 

•Acquisition  Visibility/AIR  Portal 

(https://www.dodtechipedia.mil/dodc/pluqins/AIR/airdocuments.action) 

•Acquipedia  https://dap.dau.mil/acquipedia 
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Joint  Capabilities  Integration  ft  Development  System  (JC1DS) 

JCtOS  -i  defined  m  CJCSt  3l?0.0lM.  There  ere  three  key  c'oc«5«  ,n  the  000  that  must 
work  m  concert  to  deliver  the  capabilities  required  by  the  warfighter.... 

Initial  Capabilities  Document  (ICO) 

The  Initial  Capabtabes  Document  (ICO)  documents  the  DoO  need  ter  a  materiel  approach  (or 
an  approach  that  combines  metenel  end  non  mete"e»  solution  sets)  to... 

Analysis  ol  Alternatives  ( AoA) 

The  AoA  *nrun  potential  materiel  solution*  to  satisfy  the  capability  need  documented  «n  the 

approved  Ineial  Capabilities  document  (ICO).  It  focuses  on  idem. .. 

Capability  Development  Document  (CDD) 

A  document  that  captures  the  information  necessary  to  develop  a  proposed  pregrem(s), 
normally  using  an  evolutionary  acquisition  (FA)  strategy.  The  COO  outlines  ... 

Acquisition  Ditnyory  (  At  .A  I  ) 

Categories  established  to  faoUate  decent rskied  decision  making  and  execution  and 
compliance  with  statutorky  imposed  requirements.  The  categories  determine  ... 

Test  A  F valuation  Master  Plan  (IFMP) 

Oau  Glossary:  The  timp  documents  the  overall  structure  end  objectives  of  the  Test  end 
(valuation  (T6£)  program,  it  provides  a  framework  wCtwi  whaeh  to  generate... 

Net- Ready  Key  Performance  Parameter  (KPP) 

Ooeumems  sponsor  -demified  end  Jomt  Requirement  Oversight  Council  (JROC)  vacated 

venfieWe  performance  measures  end  metrics  for  interoperebiity  engm... 

DOTmlPr  P  Analysis 

OOTmlPF  P  it  the  Ood  acronym  that  pen  am*  to  the  eight  pe 
involved  •"  solving  warfighting  capability  gap*.  These  SoMion*  may  r. 

Acquisition  Proqram  Doseline  (APR) 

From  the  OAU  Glossary  Baseline  lhal  reflect*  the  threshold 
minimum  number  of  cast,  schedule,  and  performance  attributes  that  dtaK... 

Concept  ol  Operations  (CONORS) 

A  verbal  or  graphic  statement  that  dearly  and  ccnosely  expresses  what  the  jomt  force 

how  e  will  ba  done  using  available  .. 


Operations  ft  Maintenance  F units 

Operation*  end  Hairlrrwvnr  (O&M)  fund*  include  many  asperate  appropriation*  (*g., 
Operations  and  Maintenance  Army,  Operations  and  Maintenance  Navy,  Operation*  .. 

Research  Development  Test  ft  Fvaluation  (RDTftF)  Funds 


«/2« on  toa.:«i 


8.2320 1 3  101.245 


8/28/2012  100.044 


12.22  2009  >8.034 


1/4/2013  99.4S0 


■  •2013  91.213 


#.22/2013  03, 919 


1.24  2011  82.924 


■.’24/2011  81.324 


9  24.2013  39.094 


2/14/2011  99.491 
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Automatic  Data  Streaming 
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TEMP:  Test  &  Evaluation  Master  Plan 
SEP:  Systems  Engineering  Plan 


•  Difficult  and  follow 
AVP’s  proper  request 
processes  and  rules 


ASR:  Acquisition  Strategy  Report 
SAR:  Selected  Acquisition  Report 
DAES:  Defense  Acquisition  Executive 


Summary 

ADM:  Milestone  B  2366b  Certification 
Acquisition  Decision  Memorandum 

APB:  Acquisition  Program  Baseline 

TRA:  Technology  Readiness  Assessment 

LCSP:  Life  Cycle  Sustainment  Plan 

Acquipedia 


LLA  Results/Reports: 

1 .  Match  Matrix  Report 
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A  match  score  for  a  data  source  is 
the  total  number  of  matched 
features  (e.g.,  LLA  word  pairs) 


0.15  is  the  correlation  among  these 
categories  between  Acquipedia  and 
ASR,  normalized  using  the  match  score 
and  uniqueness  score,  computed  as 
=499/  (943+3944 )  x  832+1415) 


499  is  the  number  of  word 
pairs  matched  between  two 
sources:  Acquipedia  and 
ASR 


A  uniqueness  score  is  the  total 
number  of  unique  word  pairs  are 
unique  to  the  source. 
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2.  Correlations  Among  Data  Sources 


DiSE 


Ten  Data  Sources 

o  TEMP 
o  TRA 
o  SEP 
o  SAR 
o  ADM 
o  LCSP 
o  DAES 
o  ASR 
o  APB 
o  Acquipedia 

Red  -  high  correlation 
Blue  -  low  correlation 
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Correlation  >  0.1 
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Data  sources  LCSP,  ASR  and  SEP 
have  the  highest  correlations  with 
Acquipedia  and  with  each  other. 

ASR,  LCSP  and  SEP  may  use  more 
standardized  vocabularies  and 
terminologies  than  other  data  sources. 

SAR  and  DAES  are  also  correlated 
with  each  other  more  with  each  other. 


Acquisition  Research  Program:  Creating  Synergy  for  Informed  Change 


NpVrt]  i‘;id  LUJIc  ScJl(M)l 
MtnHcrfv, 


3.  Drill-down 
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Frequencies/probability  of  the  word  pairs 


Click  the  link  to  search/drill  down  to  the  original  documents 


(in  Theme  1"2, Popularity) 
i  (in  Theme  46, Emerging) 
t  r6.Popularity) 

;  1  OS,  Anomaly) 

■  C«  Theme  46, Emerging) 
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[5$  00.0  20)KEY  SYSTEM  (in  Theme  99, Emerging) 
[37.0Q.Q.24]CQM\ERCIAL  ITEMS  (in  Theme  91, Anomaly) 

156  00.1  00|PRQGRAM  EXECl'TIM  (in  Theme  1"6, Popularity) 


Related  concepts  visualized 


From  Reports  of 
frequencies,  bi-gram 
probabilities,  categories  of 
themes 

Allows  reach-back  search 
to  original  documents 

Provides  related  concepts 
to  become  visualized 
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4.  Discovered  Theme  Report:  A  List  of  Themes/Clusters  of  Word 
Pairs  When  Comparing  Two  Data  Sources,  e.g.,  ASR  and 
Acquipedia 
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m ) 

108(A) 

20 

86 

22 

0.17 

0.83 

108(A) 

32(A) 

11 

61 

6 

0.08 

0.92 

32(A) 

149(A) 

10 

81 

11 

0.11 

0.89 

mm 

22(A) 

21 

68 

8 

0.08 

0.92 

2m 

166(A) 

16 

81 

10 

0.09 

0.91 

166(A) 

131(A) 

7 

80 

6 

0.06 

0.94 

131(A) 

115(A) 

12 

106 

3 

0.02 

0.98 

115(A) 

66(A) 

10 

59 

4 

0.05 

0.95 

66(A) 

68(A) 

16 

58 

16 

0.18 

0.82 

68(A) 

145(A) 

23 

70 

7 

0.07 

0.93 

145(A) 

Number  of  unique  features  for  Source  1  (e.g., 
ASR) 

Number  of  unique  features  for  Source  2  (e.g., 
Acquipedia) 

Matched  features  for  both  sources 

Correlation  of  two  sources  (or  consensus 
rate),  i.e.,  percentage  of  the  features  that  are 
matched 

Gap  rate:  percentage  of  the  features  that  are 
not  matched 

These  statistics  show  where  the  two  data 
sources  agree  or  disagree  the  most  (reflected 
in  the  themes) 

•  Consensus,  e.g.,  46(E) 

•  Disagreement/gap/inconsistency,  e.g., 
167(F) 

•  Clicking  on  the  Visualization  column 
of  46(E)  and  167(P)  lead  to  the 
visualizations  of  two  areas  where 
consensus  and  gap  took  place. 
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5.  One  Theme  Detail  View  for  Data  Source*  V 

DiSE 


LCSP  and  ASR 


Red  nodes  show  the  most  "central" 
nodes,  used  as  keywords  to  summarize 
this  theme,  i.e.  “Additional.Program, 
Cost,  Test,” 

Red  Links  show  word  pairs  shared  by 
the  two  sources. 


Yellow  Links  show  the  unique  word 
pairs  from  one  source  (e.g.  LCSP) 

Green  links  show  word  pairs  from  the 
other  source  (e.g.,  ASR). 


o  The  actual  word  pairs  are 

eliminated  here  since  the  content 
is  FOUO. 

The  consensus  rate  for  this  theme  is 
29%,  i.e.  29%  of  word  pairs  or  features 
are  in  agreement 

71%  of  word  pairs  are  not.  As  one  can 
see,  ASR  focuses  on  “Test”  and  LCSP 
does  not. 
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6.  Report  the  List  of  Word  Pairs  Matched  and  Unique  in 
the  Two  Data  Sources  (e.g.,  LCSP  and  ASR) 


S  % 

DISE 


•  nod<i:  the  most  ‘centraT  nodes,  used  m  keywords  to  svnmenze  ttw  theme 

•  R«»d  u nis  word  pairs  by  at  least  two  sources 

•  Green  links  word  pars  from  Source 

Total  Consensus  Features  for  LCSP, ASR  26 
Total  Uraquness  Features  for  LCSP:  14 

Total  Urwquness  Features  for  ASR:  49 

Gap  Rate:  0.71 

Consensus  Rate:  0.29 

Consensus  Features  for  LCSP.ASR 

Unique  Features  for  ASR 

[11 

[2] 

[2K 

[2] 

[3] 

[611 

[6] 

[61 

[231 

[12 

[21 

[21: 

[11 

[23 

[211 

[12 

[23 

[23 

[3] 

[23 

[21! 

[21 

[61 

[41: 

[68 

[2] 

[411 

[30 

!61 

[63  r 

[3)f 

[21 

[31- 

171' 

[21 

[33- 

[10 

[2]  ) 

[2]E 

[6]‘ 

[31 

[33 

in 

[2)r 

[2] 

(4p. 

[5] 

[21- 

[15 

[21 

[  2  j 

[81  ■ 

[3J* 

[2] 

[23S 

[3K 

[2] 

[4] 

[2] 

[21 

[3: 

[2] 

[13 

[2! 

[12.  .  . 

[31- 

[21- 

[215 

[2)5 

[3] 

[21- 

[4)0 

[2]: 

[21! 

[21: 

[211 

[31. 

[21: 

[31. 

The  actual  word 
pairs  are  eliminated 
here  since  the 
content  is  for 
official  use  only. 
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Acquipedia:  LLA  generated  910  word  pairs  (out  of  5062  total  word  pairs) 
matched  from  this  ship-building  program’s  nine  data  sources.  The  list 
can  be  considered  as  a  set  of  standardized  vocabularies  and 
terminologies 
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Appendix  A:  Word  Pairs  in  Acquipedia  Matched  in  a  Ship 
building  Program 

r682.00.0.341PRQGRAM  MANAGEMENT  (in  Theme  1  76.Popularitv) 
f3  1  8.00.0.331  SYSTEMS  ENGINEERING  (in  Theme  lOS.Anomalv) 
r280.00.Q.921PROGRAM  OFFICE  (in  Thane  131.Anomalv) 

[24 1 .00.0. 601TECHNI  CAL  DATA  (in  Theme  46.EmerEinE) 
f  1 80.00.1  .OOIPROGRAM  MANAGERS  Theme  lOS.Anomalv) 
f  180. 00.0. 241UFE  CYCLE  MANAGEMENT  (in Theme  lOS.Anomalv’) 

[1 73 .00.0 . 131 ACQUISITION  MANAGEMENT  (in  Theme  1 5 7.Popukritv) 
f  163. 00.0. 361JOINT  CAPABILITIES  (in  Theme  67.Anomalv) 
ri6Q.00.0.541PRQDUCT  SUPPORT  MANAGER  (in  Theme  lOS.Anomalv) 
ri59.00.0.281LIFE  CYCLE  SUSTAINMENT  (in  Theme  1 5 7. Popularity) 
f  147. 00.0. 251 ACOLTSITION PROGRAM  (in Theme  157,Popularity) 
ri22.00.0.441LOGISTIC.S  SUPPORT  (in  Theme  169.EmersinE) 
n  19.00.0. 321LEUEL  MAINTEN.ANCE  (in  Theme  106.Anomalv) 
n  12. 00.0. 061  SYSTEM  MANAGEMENT  (in  Theme  172.Popularitv) 

[1 1 1 .00. 1 .001PERFQRMANCE  INDEX  (in  Theme  46.EmerEina) 
ri08.00.0.351RISK  MANAGEMENT  (in  Theme  176.Pomilaritv'i 
UPS. 00.0 .631MANAGERPSM (in Theme  lOS.Anomalv) 

1103. 00. 1.001INFORMATI ON  ASSURANCE  (in  Theme  46.Emersing) 
ri02.00.0.201TEST  EVALUATION  (in  Theme  104.Emerein£) 
ri02.00.0.311INTE GRATED  PROGRAM  (in  Theme  22.Anomaly) 
r99.00.d.371GQ\’ERNMENT  PROPERTY  (in  Theme  7 S. Anomaly) 

182.00. 1.001CQSTESTIMATING(in  Theme  176:Popularity) 
r80.00.0.471DATARIGHTS  (in  Theme  46,EmerEins) 

P9.00.0. 181  SUPPORT  EQUIPMENT  (in  Theme  6~.Anomalv) 

P8.00.0.821WEAPQN  SYSTEMS  (in  Theme  172.Popularitv) 

176.00.0. 181LIFE  CYCLE  COST  (in  Theme  lOS.Anomalv) 

P 5 .00.0.3 71PL.ANNING PACKAGE  (in  Theme  120.EmeiEing) 

P3.00.0.211MANAGEMENT  OFFICE  (in  Theme  131. Anomaly) 

P3.00.0.251MAINTEN.ANCE  PLANNING  (in  Theme  120.EmerEins) 


P2.00.0.0 71 REQUIREMENTS  MANAGEMENT  (in  Theme  154.Emer:hng) 
P0.00.0.191MANAGEMENT  IMPLEMENTATION  (in  Theme  1  lS.An'omalv) 
r68.00.0.101CQNTRACT  PERFORMANCE  (in  Theme  46  .Emerging) 
r6S. 00.0. 141  SYSTEM  PERFORMANCE  (in  Theme  46, Emerging 
r6S.00.0.081DATAhLANAGEMENT  (in  Theme  46.Emerging) 

[6S.00'l  .001j.LaNLTaCTURINGDEA~ELOPMENT  (in  Theme  1 5  7!Popularity) 


Distribution  of  Acquipedia  Features 
Among  the  910  Matched  Features 


m  % 
2  % 

DtSE 


39% 

35% 

25% 


358 

322 

230 


Popular  or  Normal  (P): 

•  Themes  contain  the  highest  number  of  mutually  connected  word  pairs. 

•  Themes  represent  the  main  topics  in  a  corpus  at  the  time. 


This  validates  that  our 
observations  :  Anomalous  and 
emerging  features  are  more 
interesting  because  they  are 
used  in  the  documents 
regarding  an  actual  ship¬ 
building  program. 


•  May  be  also  regarded  as  less  interesting  because  they  are  already  in  the  public 
consensus  and  awareness,  therefore,  less  room  for  growth. 

Emerging  (E):  themes  containing  the  medium  number  of  mutually  connected  word  pairs, 
these  themes  may  grow  to  popular  over  time  as  we  show  later  in  the  examples. 

Anomalous  (A):  themes  containing  the  lowest  number  of  mutually  connected  word  pairs. 
These  themes  may  be  off-topics  which  may  seem  they  do  not  belong  here  compared  to 
other  ones  and  may  be  interesting  for  further  investigation. 
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Summary 


•  LLA  was  used  to  analyze  an  MDAP  program  using  AVP 
data  in  which  we 


-  Demonstrated  a  set  of  comprehensive  LLA  analysis  reports  and 
visualizations  generated  automatically  using  multiple  categories  of 
program  data  as  data  sources. 


-  Revealed  correlations  and  gaps  among  at  least  eight  data  sources. 

-  Formed  the  basis  for  further  inquiry  or  future  reconciliation  of  the 
expectations  (e.g.,  acquisition  strategy)  and  realities  (e.g.,  engineering 
feasibility)  from  various  communities  for  the  same  MDAP  program. 


-  Discovered  in  detail  where  the  gaps  and  inconsistencies  of  the  data 
across  multiple  data  sources  reside  which  lead  to  the  identification  of 
future  specific  and  productive  directions  for  further  examination 
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Planned  Future  Work 
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•  Continuing  work  with  Sponsors  and  AVP  analysts  to 
develop  a  process  to  generate  the  LLA  reports  and 
visualizations  for  any  given  program  in  AVP. 

•  Studying  the  program  interactions  for  a  portfolio  of 
programs. 


-  Select  a  portfolio  of  programs  and  focuses  on  one 
type  of  data  sources,  for  example  ASR  to  see  how 
LLA  can  depict  the  interaction  risks. 


•  Conducting  supervised  learning  data  to  train  LLA 
using  Acquipedia  to  improve  the  understanding  of 
context-dependent  meaning. 
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